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VIDEO COMFORT NOISE ADDITION TECHNIQUE 

CROSS-REFERENCE TO RELATED APPLICATIONS 

This application claims priority under 35 U.S.C. 1 19(e) to U.S. Provisional Patent 
Application Serial No. 60/505,254 filed on September 23, 2003, the teachings of which are 
incorporated herein. 
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TECHNICAL FIELD 

This invention relates to a technique for reducing artifacts in connection with decoding 
of a coded video stream. 



BACKGROUND ART 

15 

The decoding of a video stream compressed at low bit rate often yields visible artifacts 
noticeable to a viewer. Blockiness and structured noise patterns are common artifacts that 
arise when using block-based compression techniques. The human visual system has a 
greater sensitivity to certain types of artifacts, and thus, such artifacts appear more noticeable 
20 and objectionable than others. The addition of random noise to the decoded stream can 

reduce the noticeability of such compression artifacts, but large frame-to-frame differences 
created by adding random noise can itself produce artifacts that appear noticeable and 
objectionable. 

The addition of a dither signal can reduce human sensitivity to image artifacts, for 
25 example to hide contouring and blocking artifacts. One prior art technique has proposed 

adding a random noise dither that is based on film grain to an image to disguise block effects. 
The rationale for adding such random noise is that random error is more forgiving than the 
structure, or correlated error. Other prior art techniques have proposed adding a dither signal 
to a video stream to hide compression artifacts. One past technique has proposed adding a 
30 random noise dither in the video encoding and decoding process in the in loop deblocking 
filter for the ITU/ISO H. 264 video coding standard, commonly known as the JVT coding 
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standard. The amount of dither to be added depends on the position of a pixel with respect to 
a block edge. Another prior technique has proposed adding that random noise subsequent to 
video decoding (i.e., adding noise as a "post process"), for use as comfort noise. The amount 
of noise added depends on the quantization parameter and on the amount of noise added to 

5 spatially neighboring pixels. The term "comfort noise" comes from the use of noise in audio 
compression to indicate noise pattern generated at the receiver end to avoid total silence that is 
uncomfortable to a listener. 

Past techniques for reducing artifacts by adding noise typically reduce spatial artifacts 
at the risk of creating temporal abnormalities, i.e., large frame-to-frame differences. Thus, 

10 there exists a need for a technique for reducing artifacts during decoding of a coded video 
stream that overcomes the aforementioned disadvantages. 

BRIEF SUMMARY OF THE INVENTION 

1 5 Briefly, in accordance with a preferred embodiment of the present principles, a method 

is provided for reduced artifacts in a video stream during decoding. The method commences 
by decoding the video stream. Following decoding, noise is added to the video stream by 
adding noise to each pixel in an amount correlated to luminance of at least a portion of a 
previously decoded picture. Thus, in accordance with the present principles, luminance 

20 correlation aids in determining the additive noise to reduce large frame-to-frame differences, a 
disadvantage of prior noise additive techniques. 

BRIEF DESCRIPTION OF THE DRAWINGS 

25 FIGURE 1 depicts a block schematic diagram of a first embodiment of a video 

decoder arrangement in accordance with the present principles for reducing artifacts in 
connection with decoding a coded video stream by adding noise correlated to the luminance 
of at least a portion of the current picture; 

FIGURE 2 depicts a block schematic diagram of a second embodiment of a video 

3 O decoder arrangement in accordance with the present principles for reducing artifacts in 
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connection with decoding a coded video stream by adding noise correlated to the luminance 
of at least a portion of the current picture; and 

FIGURE 3 depicts a block schematic diagram of a third embodiment of a video 
decoder arrangement in accordance with the present principles for reducing artifacts in 
connection with decoding a coded video stream by adding noise correlated to the luminance 
of at least a portion of the current picture. 

DETAILED DESCRIPTION 

In accordance with the present principles, adding a random-noise, containing dither 
signal, to already decoded signal, in an amount correlated to the luminance of at least a 
portion of a current picture, improves the subjective video quality. 

Heretofore, adding noise to a decoded signal has been found to improve the quality of 
the video signal. The visual impact of adding a noise signal to the video sequence, rather than 
15 just to a single image, becomes a consideration in the determination of the magnitude of the 
noise signal. Consideration has been given to the visual impact of adding a noise signal to the 
video sequence, rather than just to a single image in the determination of the magnitude of the 
noise signal. The magnitude of additive noise signal for a pixel in a picture can be correlated 
to the value of the additive noise signal of the pixels in the previously displayed picture, e.g., 
20 the noise signals are temporally correlated. Alternatively, the temporal correlation can be 
based on the previously decoded picture, rather than the previously displayed picture. 

Based on the foregoing, the added noise signal, using temporal correlation with a 
correlation factor a, 0 < a < 1, can be computed as 

25 N(k, x, y) = (1 - a) * N(k-1, x, y) + a * R(k, x, y) (D 

The random number R(k, x, y) can be generated using any type of random number 
distribution, for example a Normalized, Gaussian, or Laplacian distribution. R(k, x, y) may 
also be clipped within a certain range if necessary. The random number generator may be 
30 implemented by means of a lookup table. R(k, x, y) may also include spatial correlation, such 
as that used for example in film grain noise generation. 
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In accordance with present principles, noise addition appears quite dependent on the 
brightness (i.e., luminance) of a block or macroblock, but also to that of its adjacent blocks. 
The darker the block/macroblock, the easier it becomes to notice noise with relatively high 
variance. On this basis, the amount of additive noise N(k, x, y) can be given by the 
5 relationship: 

N(k, x, y) = (1 - Y (k, x, y)) * N(k-1, x, y) + y(k, x, y) * (1 - ftk, Y» *R<k. Y> ( 2 ) 

with the function y (k, x, y) representing a correlation factor dependent on the temporal 
10 correlation of the current image with the previous displayed or decoded one. 



15 



The term y (k, x, y) can be computed as : 

Y (k, x, y)= a - P */XD(k, x, y), D(k-1, x, y)), 0 < P < a < 1 (3) 



where/}0 takes values between 0 and 1 and computes the temporal correlation factor of pixel 
(x, y) in picture k with its co-located pixel in picture k-1. The Factors a and p here relate to 
the picture type (I, P or B picture) as well as the quantizer used for coding the current picture 
or block, and can be calculated through the use of a lookup table. Alternatively, the full 
20 resolution difference image between pictures k and k-1, can be used and the two pictures may 
I be considered as correlated (i.e. fi=l) if the total absolute difference is below a value, 
'' It is also possible to consider simpler metrics. For example considerable savings in 

storage and computation can occur by considering the mean of NxN blocks instead and 
perform noise adaptation at a block level. In this case, the term /.(DQc.x, y),D(k-l, x, y)) will 

25 equal: 

^(DOc x, y),D(k-l,x, y))= {-^ab S (f J f l D(k,x+k,y+k) 

iV X i V fc=Q 171=0 



*=0 m=0 (4) 

D(k - Ux + k, y + *)) > Co ?0 : D 

(=Ot 



where 0< ^>^255. 
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The Term <))(k, x, y) reflects spatial information to adjust the strength of the noise that will be 
used. In particular <|>(k, x, y) can be computed as: 

5 <|>(k, x, y) =/ 2 (D(k, x, y)) 

+/ 5 (D(k, x-bsx, y), D(k, x+ bsx, y), D(k, x, y- bsy) , D(k, x, y+bsy)) (4) 

where f 2 Q relates to the brightness of the current pixel or the N x N block to which it belongs, 
while /i0 computes the spatial relationship between the current pixel/block with it's 
10 horizontally or vertically adjacent at a distance of bsx or bsy. For example, 

f 2 (D(k, x, y)) = D <*' * + k,y + k)>C l 10:l) (5) 

/ 3 (D(k, x, y)) = (/ 2 (D(k, x, y)) - / 2 (D(k, x + N, y)) = £ 2 ) II (/ 2 (D(k, x, y)) - / 2 (D(k, x - N, y)) = £,) 
II (/ 2 (D(k, x, y)) - / 2 (D(k, x, y + N)) = ^) II (/ 2 (D(k, x, y)) - / 2 (D(k, x, y - N)) = f 5 ) 



where 0<^ <255, and -255<^ ^, £ ^255. 

15 

FIGURE 1 depicts a block schematic diagram of a first embodiment of a video 
decoder arrangement 10 for adding noise correlated to the luminance of at least a portion of 
the current picture in a manner compatible with Equation 2 to reduce artifacts. The decoder 
arrangement 10 includes a decoder 12 for decoding an incoming coded video stream. The 

20 design of decoder 12 depends on the compression format employed to code the incoming 

video stream. In a preferred embodiment, the incoming video stream undergoes compression 
using the well-known ITU/ISO H. 264 standard. Under such circumstances, the decoder 12 
takes the form of a H.264 decoder known in the art. A reference picture store 14 stores 
pictures decoded by the decoder 12 for use by the decoder in decoding future pictures. 

25 The decoder 12 supplies a noise generator 16 with both a decoded picture, as well as 

bit stream information contained in the decoded picture. The bit stream information output by 
the decoder 12 can include a quantization parameter input to the noise generator. The severity 
of compression artifacts is correlated to the quantization parameter, with more severe 
compression artifacts occurring when high quantization parameter values are used. The 
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strength of the added comfort noise can be increased as the quantization parameter value 
increases. 

A summing block 18 sums each decoded picture from the decoder 12 with noise from 
a noise generator 16. A clipper 20 then clips the resultant signal output by the summing block 
18 to yield a decoded picture for display which exhibits reduced artifacts. Note that noise 
addition occurs after storage of decoded pictures in the reference picture store 14 since the 
reference pictures must remain unchanged in order to properly decode the subsequent 
incoming pictures. 

A noise picture store 17 stores the noise signal N(k, x, y) for the k & picture for 
subsequent use by the noise generator 16. The noise generator 16 responds to reference 
pictures stored in the reference picture 14 store, which contains information about previously 
decoded pictures. Although not necessary, an additional storage could be added if block 
based computation for the temporal correlation between decoded pictures is used. 

While noise generation for each pixel within an image remains possible, in certain 
cases (i.e. for higher resolution material), generation of larger size (grain) noise often proves 
more desirable. For example, applying an N x N block size Discrete Cosine Transform on the 
noise image, and then discarding the resultant higher frequencies will yield a larger size noise 
similar to film grain noise. This process nevertheless incurs a relatively large expense and 
typically will require a deblocking process in order to reduce blocking artifacts that might be 
generated on the block edges. 

FIGURE 2 depicts a block schematic diagram of a second embodiment of a video 
decoder arrangement 100 for adding large grain noise correlated to the luminance of at least a 
portion of the current picture. The decoder arrangement 100 includes many of the same 
elements as the decoder arrangement 10 of FIG. 1, and like reference numbers identify like 
elements. As compared to the decoder arrangement 10 of FIG. 1, the decoder arrangement 
100 of FIG. 2 further includes a N x N reduced picture average store 22 coupled to the 
reference picture store 14. The picture store 22 typically stores NxN luma block average 
values. The average luma values stored in the picture store 22 allow the decoder arrangement 
to generate larger grain noise as discussed. 

FIGURE 3 depicts a block schematic diagram of a third embodiment of a video 
decoder arrangement 1000 for adding large grain noise correlated to the luminance of at least 
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a portion of the current picture. The decoder arrangement 1000 of FIG. 3 includes many of 
the same elements as the decoder arrangement 100 of FIG. 2, and like reference numbers 
identify like elements. As compared to the decoder arrangement 100 of FIG. 2, the decoder 
arrangement 1000 of FIG. 3 contains no noise picture store 17, but only the N x N reduced 
5 picture average store 22. 

An alternative and considerably simpler process would be to generate the noise at a 
smaller resolution than that of the original image (e.g. half horizontal and vertical resolution), 
and then up sample the noise (e.g. using sample replication). Using the original or smaller 
resolution could also be decided based on the resolution of the original pictures (e.g. use same 
10 resolution for Standard Definition and lower definition material, while using lower resolution 
noise generation for High Definition material). Side parameters could also be transmitted 
with the bit stream that would allow the decoder to decide which process shall be used. Side 
information could also be used for the generation of noise (e.g. noise variance weighting). 
This exact same process could also be applied onto color components as well. 
15 Nevertheless, to reduce complexity and computation, noise generation could occur based only 
on one luminance component (i.e. luma), and while re-using the same noise on all color 
components, after performing a simple scaling and sub-sampling if necessary. For example, 
for 4:2:0 material, the luma noise is vertically and horizontally sub-sampled by 2, and can be 
divided by 2 in order to generate chroma noise. 
20 The decoder arrangement 10 and 100 of FIGS. 1 and 2 represent instantiations of a 

temporal Infinite Impulse Response (DR) filter. The IIR filter may be generalized by using 
more filter taps. IIR filters can also generally be approximated using higher order FIR filters, 
using as many taps, t, as desired in accordance with the following relationship: 

N(k,x, y) = JJQl - y{k - j,x, y))x N(k - 1, x, y) 

+jnri(i-rt*-j.^ 

25 A Finite Impulse Response (FIR) filter approach can be implemented using the 

decoder arrangement of FIGURE 3. The decoder arrangement 1000 only makes use of the 
previous random numbers R, and if necessary, the N x N luma block mean values, rather than 
the previous noise N, in such an FIR filter approach, thus reducing memory bandwidth. It is 
possible also to use and store only the N x N luma block average values of the current and 
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previous picture, and reuse the same values and their difference for all taps. For example we 
may use the following system: 

N(k,x, y) = (1 - y(M> x, y)) x (1 - y(k - 1, x 9 y» x R(k - 2, x, y) 

+ (1 - Y{k,x, y)) x y(k - 1, jc, 3>)x (1 - $(k - 1.x, y)) x - 1, x 9 y) (8) 
+ x, y) X ( 1 - <p(K y)) X R(k,x, y) 

although it is further possible to simplify the above by forcing the difference images used in 
the computation of y{k -l,x,y) to be the same as that of y(k t x,y) . This would completely 
avoid the need to store or re-compute the difference image, and considerably reduce memory 
bandwidth. 

The foregoing describes a technique for reducing artifacts in connection with decoding 
of a coded video stream by adding noise correlated to the luminance of at least a portion of the 
current picture. 



